Skip to content

feat(#192): add evidence cap to retro-analysis skill - #305

Closed
fullsend-ai-coder[bot] wants to merge 1 commit into
mainfrom
agent/192-cap-evidence-issues
Closed

feat(#192): add evidence cap to retro-analysis skill#305
fullsend-ai-coder[bot] wants to merge 1 commit into
mainfrom
agent/192-cap-evidence-issues

Conversation

@fullsend-ai-coder

@fullsend-ai-coder fullsend-ai-coder Bot commented Jul 12, 2026

Copy link
Copy Markdown

Add an evidence cap to the retro-analysis skill's "Before proposing" section. Before filing an evidence proposal for a parent issue, the retro agent now queries the count of open evidence issues for that parent. If the count is >= 5, the agent skips the proposal and instead notes the parent issue and current PR/issue as a data point in the summary comment.

This prevents the retro agent from filing redundant evidence issues for well-documented patterns, reducing tracking noise in repos with many open evidence issues.

Addresses all review findings from prior PR #211:

  • Explicit fail-open guidance with summary observability
  • Enforced title prefix convention for evidence issues
  • Clarified ordering relative to the duplicate check
  • Trailing colon in query to reduce false positives
  • Bold heading to match section conventions

Note: pre-commit could not run in sandbox (shellcheck install failed due to network restrictions). The post-script runs pre-commit authoritatively on the runner.

Closes #192

Post-script verification

  • Branch is not main/master (agent/192-cap-evidence-issues)
  • Secret scan passed (gitleaks — a2afa8b179a351845733c680b839a39a82304db6..HEAD)
  • Pre-commit hooks passed (authoritative run on runner)
  • Tests ran inside sandbox

@fullsend-ai-coder
feat(#192): add evidence cap to retro-analysis skill
9865a6d 
Add an evidence cap to the retro-analysis skill's "Before proposing"
section. Before filing an evidence proposal for a parent issue, the
retro agent now queries the count of open evidence issues for that
parent. If the count is >= 5, the agent skips the proposal and
instead notes the parent issue and current PR/issue as a data point
in the summary comment.

This prevents the retro agent from filing redundant evidence issues
for well-documented patterns, reducing tracking noise in repos with
many open evidence issues.

Addresses all review findings from prior PR #211:
- Explicit fail-open guidance with summary observability
- Enforced title prefix convention for evidence issues
- Clarified ordering relative to the duplicate check
- Trailing colon in query to reduce false positives
- Bold heading to match section conventions

Note: pre-commit could not run in sandbox (shellcheck install
failed due to network restrictions). The post-script runs
pre-commit authoritatively on the runner.

Closes #192
@github-actions

github-actions Bot commented Jul 12, 2026

Copy link
Copy Markdown

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 12, 2026 •
edited
Loading

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 3:39 AM UTC · Completed 3:47 AM UTC
Commit: a2afa8b · View workflow run →

@fullsend-ai-review

fullsend-ai-review Bot commented Jul 12, 2026

Copy link
Copy Markdown

Review — PR #305

Verdict: Approve

Scope: internal/scaffold/fullsend-repo/skills/retro-analysis/SKILL.md (+56/−0)
Authorization: Issue #192 — Cap evidence issues per parent

Summary

This PR adds an evidence cap section to the retro-analysis skill's "Before proposing" area. The change is well-scoped, architecturally correct, and cleanly implements the feature requested in #192. The new section instructs the retro agent to query the count of open evidence issues for a parent issue and skip filing if the count is ≥ 5, with fail-open behavior on query errors.

No blocking findings. Three low-severity observations are noted below for awareness.

Findings

1. in:title means "contains," not "starts with" — low · correctness
The prose at the "Dispatch a subagent" section says the query counts issues "whose title starts with 'Evidence for #N:'," but GitHub's search API in:title qualifier restricts where keywords match, not the position within the title. An issue titled "Revisiting Evidence for #42: ..." would also match. This is functionally harmless since the Evidence for #N: prefix convention established by this same section makes mid-title matches extremely unlikely, and slight overcounting is safe (it would suppress proposals sooner, not later). The prose could say "whose title contains" instead of "whose title starts with" for precision.

2. Non-numeric query result not explicitly handled — low · correctness
If the --jq '.total_count' expression returns null (e.g., unexpected API response shape), the instructions don't explicitly classify this as a failure. In practice, the agent would treat a non-numeric response as an error and fall into the fail-open path, so the behavior is correct by default. An explicit note ("Treat non-numeric results the same as query failures") would make the intent unambiguous.

3. No explicit validation of <parent_number> — low · defense-in-depth
The <parent_number> placeholder is interpolated into the search URL without an instruction to validate it as a positive integer. The value originates from the agent's own analysis of issue titles, not from external input, and the worst case for a malformed value is a search query that returns zero results (triggering the fail-open path). Adding a brief note ("Validate that <parent_number> matches ^[0-9]+$ before running the query") would align with the validation discipline applied to target_repo elsewhere in the codebase.

Checks performed

Dimension Result
Correctness ✅ Algorithm is internally consistent. Query is syntactically valid; %23 correctly encodes #. Ordering (after duplicate check) is logical. Edge cases (0, 5, errors) handled.
Security ✅ No injection vectors, no permission changes, no secrets exposure. Fail-open is appropriate — the cap is a noise-reduction guardrail, not a security boundary. No prompt injection patterns detected.
Intent & coherence ✅ Authorized by #192. Correctly placed in the skill file (not the agent definition). Scope exactly matches the issue's request. No scope creep.
Style & conventions ✅ Bold heading matches existing **Evaluation criteria** pattern. Code blocks use ```bash. Placeholders (<target_repo>, <parent_number>) follow established conventions. Imperative prose style is consistent.
Documentation currency ✅ No stale docs. agents/retro.md already defers to the skill's "Before proposing" section. User-facing docs/agents/retro.md operates at a higher abstraction level and doesn't need updating for this implementation detail.
Cross-repo contracts ✅ No schema or API changes. The retro-result.schema.json output contract is unchanged.

@fullsend-ai-review fullsend-ai-review Bot added the ready-for-merge All reviewers approved — ready to merge label Jul 12, 2026
@guyoron1 guyoron1 closed this Jul 12, 2026
@guyoron1
guyoron1 deleted the agent/192-cap-evidence-issues branch July 12, 2026 06:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@fullsend-ai-review fullsend-ai-review[bot] fullsend-ai-review[bot] approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

ready-for-merge All reviewers approved — ready to merge

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[ARCHIVED] Original issue 192

1 participant

@guyoron1